

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  <meta name="generator" content="Docutils 0.19: https://docutils.sourceforge.io/" />

  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>CRUSH MSR (Multi-step Retry) &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/css/custom.css" type="text/css" />

  
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
        <script src="../../_static/jquery.js"></script>
        <script src="../../_static/_sphinx_javascript_frameworks_compat.js"></script>
        <script data-url_root="../../" id="documentation_options" src="../../_static/documentation_options.js"></script>
        <script src="../../_static/doctools.js"></script>
        <script src="../../_static/sphinx_highlight.js"></script>
    
    <script type="text/javascript" src="../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../genindex/" />
    <link rel="search" title="Search" href="../../search/" />
    <link rel="next" title="C++17 and libstdc++ ABI" href="../cxx/" />
    <link rel="prev" title="Oprofile 的安装" href="../cpu-profiler/" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    <div role="navigation" aria-label="Page navigation">
  <ul class="wy-breadcrumbs">
      <li><a href="../../" class="icon icon-home" aria-label="Home"></a></li>
          <li class="breadcrumb-item"><a href="../internals/">Ceph 内幕</a></li>
      <li class="breadcrumb-item active">CRUSH MSR (Multi-step Retry)</li>
      <li class="wy-breadcrumbs-aside">
            <a href="../../_sources/dev/crush-msr.rst.txt" rel="nofollow"> View page source</a>
      </li>
  </ul>
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../" class="icon icon-home"> Ceph
          

          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" aria-label="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../start/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../install/">安装 Ceph</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../cephadm/">Cephadm</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../rados/">Ceph 存储集群</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../cephfs/">Ceph 文件系统</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../monitoring/">监控概览</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../architecture/">体系结构</a></li>
<li class="toctree-l1"><a class="reference internal" href="../developer_guide/">开发者指南</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../internals/">Ceph 内幕</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../balancer-design/">Ceph 如何均衡（读写、容量）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../blkin/">Tracing Ceph With LTTng</a></li>
<li class="toctree-l2"><a class="reference internal" href="../blkin/#tracing-ceph-with-blkin">Tracing Ceph With Blkin</a></li>
<li class="toctree-l2"><a class="reference internal" href="../bluestore/">BlueStore Internals</a></li>
<li class="toctree-l2"><a class="reference internal" href="../ceph_krb_auth/">如何配置好 Ceph Kerberos 认证的详细文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephfs-mirroring/">CephFS Mirroring</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephfs-reclaim/">CephFS Reclaim Interface</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephfs-snapshots/">CephFS 快照</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephx/">Cephx</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cephx_protocol/">Cephx 认证协议详细阐述</a></li>
<li class="toctree-l2"><a class="reference internal" href="../config/">配置管理系统</a></li>
<li class="toctree-l2"><a class="reference internal" href="../config-key/">config-key layout</a></li>
<li class="toctree-l2"><a class="reference internal" href="../context/">CephContext</a></li>
<li class="toctree-l2"><a class="reference internal" href="../continuous-integration/">Continuous Integration Architecture</a></li>
<li class="toctree-l2"><a class="reference internal" href="../corpus/">资料库结构</a></li>
<li class="toctree-l2"><a class="reference internal" href="../cpu-profiler/">Oprofile 的安装</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">CRUSH MSR (Multi-step Retry)</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#motivation">Motivation</a></li>
<li class="toctree-l3"><a class="reference internal" href="#msr">MSR</a></li>
<li class="toctree-l3"><a class="reference internal" href="#rule-structure">Rule Structure</a></li>
<li class="toctree-l3"><a class="reference internal" href="#working-space">Working Space</a></li>
<li class="toctree-l3"><a class="reference internal" href="#implementation">Implementation</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../cxx/">C++17 and libstdc++ ABI</a></li>
<li class="toctree-l2"><a class="reference internal" href="../deduplication/">去重</a></li>
<li class="toctree-l2"><a class="reference internal" href="../delayed-delete/">CephFS delayed deletion</a></li>
<li class="toctree-l2"><a class="reference internal" href="../dev_cluster_deployment/">开发集群的部署</a></li>
<li class="toctree-l2"><a class="reference internal" href="../dev_cluster_deployment/#id5">在同一机器上部署多套开发集群</a></li>
<li class="toctree-l2"><a class="reference internal" href="../development-workflow/">开发流程</a></li>
<li class="toctree-l2"><a class="reference internal" href="../documenting/">为 Ceph 写作文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../dpdk/">Ceph messenger DPDKStack</a></li>
<li class="toctree-l2"><a class="reference internal" href="../encoding/">序列化（编码、解码）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../erasure-coded-pool/">纠删码存储池</a></li>
<li class="toctree-l2"><a class="reference internal" href="../file-striping/">File striping</a></li>
<li class="toctree-l2"><a class="reference internal" href="../freebsd/">FreeBSD Implementation details</a></li>
<li class="toctree-l2"><a class="reference internal" href="../generatedocs/">Ceph 文档的构建</a></li>
<li class="toctree-l2"><a class="reference internal" href="../health-reports/">Health Reports</a></li>
<li class="toctree-l2"><a class="reference internal" href="../iana/">IANA 号</a></li>
<li class="toctree-l2"><a class="reference internal" href="../kclient/">Testing changes to the Linux Kernel CephFS driver</a></li>
<li class="toctree-l2"><a class="reference internal" href="../kclient/#step-one-build-the-kernel">Step One: build the kernel</a></li>
<li class="toctree-l2"><a class="reference internal" href="../kclient/#step-two-create-a-vm">Step Two: create a VM</a></li>
<li class="toctree-l2"><a class="reference internal" href="../kclient/#step-three-networking-the-vm">Step Three: Networking the VM</a></li>
<li class="toctree-l2"><a class="reference internal" href="../kubernetes/">Hacking on Ceph in Kubernetes with Rook</a></li>
<li class="toctree-l2"><a class="reference internal" href="../libcephfs_proxy/">Design of the libcephfs proxy</a></li>
<li class="toctree-l2"><a class="reference internal" href="../libs/">库体系结构</a></li>
<li class="toctree-l2"><a class="reference internal" href="../logging/">集群日志的用法</a></li>
<li class="toctree-l2"><a class="reference internal" href="../logs/">调试日志</a></li>
<li class="toctree-l2"><a class="reference internal" href="../macos/">在 MacOS 上构建</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mempool_accounting/">What is a mempool?</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mempool_accounting/#some-common-mempools-that-we-can-track">Some common mempools that we can track</a></li>
<li class="toctree-l2"><a class="reference internal" href="../messenger/">Messenger notes</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mon-bootstrap/">Monitor bootstrap</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mon-elections/">Monitor Elections</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mon-on-disk-formats/">ON-DISK FORMAT</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mon-osdmap-prune/">FULL OSDMAP VERSION PRUNING</a></li>
<li class="toctree-l2"><a class="reference internal" href="../msgr2/">msgr2 协议（ msgr2.0 和 msgr2.1 ）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../network-encoding/">Network Encoding</a></li>
<li class="toctree-l2"><a class="reference internal" href="../network-protocol/">网络协议</a></li>
<li class="toctree-l2"><a class="reference internal" href="../object-store/">对象存储架构概述</a></li>
<li class="toctree-l2"><a class="reference internal" href="../osd-class-path/">OSD class path issues</a></li>
<li class="toctree-l2"><a class="reference internal" href="../peering/">互联</a></li>
<li class="toctree-l2"><a class="reference internal" href="../perf/">Using perf</a></li>
<li class="toctree-l2"><a class="reference internal" href="../perf_counters/">性能计数器</a></li>
<li class="toctree-l2"><a class="reference internal" href="../perf_histograms/">Perf histograms</a></li>
<li class="toctree-l2"><a class="reference internal" href="../placement-group/">PG （归置组）说明</a></li>
<li class="toctree-l2"><a class="reference internal" href="../quick_guide/">开发者指南（快速）</a></li>
<li class="toctree-l2"><a class="reference internal" href="../rados-client-protocol/">RADOS 客户端协议</a></li>
<li class="toctree-l2"><a class="reference internal" href="../rbd-diff/">RBD 增量备份</a></li>
<li class="toctree-l2"><a class="reference internal" href="../rbd-export/">RBD Export &amp; Import</a></li>
<li class="toctree-l2"><a class="reference internal" href="../rbd-layering/">RBD Layering</a></li>
<li class="toctree-l2"><a class="reference internal" href="../release-checklists/">Release checklists</a></li>
<li class="toctree-l2"><a class="reference internal" href="../release-process/">Ceph Release Process</a></li>
<li class="toctree-l2"><a class="reference internal" href="../seastore/">SeaStore</a></li>
<li class="toctree-l2"><a class="reference internal" href="../sepia/">Sepia 社区测试实验室</a></li>
<li class="toctree-l2"><a class="reference internal" href="../session_authentication/">Session Authentication for the Cephx Protocol</a></li>
<li class="toctree-l2"><a class="reference internal" href="../testing/">测试笔记</a></li>
<li class="toctree-l2"><a class="reference internal" href="../versions/">Public OSD Version</a></li>
<li class="toctree-l2"><a class="reference internal" href="../vstart-ganesha/">NFS CephFS-RGW Developer Guide</a></li>
<li class="toctree-l2"><a class="reference internal" href="../wireshark/">Wireshark Dissector</a></li>
<li class="toctree-l2"><a class="reference internal" href="../zoned-storage/">Zoned Storage Support</a></li>
<li class="toctree-l2"><a class="reference internal" href="../osd_internals/">OSD 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mds_internals/">MDS 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../radosgw/">RADOS 网关开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../ceph-volume/">ceph-volume 开发者文档</a></li>
<li class="toctree-l2"><a class="reference internal" href="../crimson/">Crimson developer documentation</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../hardware-monitoring/">硬件监控</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <section id="crush-msr-multi-step-retry">
<h1>CRUSH MSR (Multi-step Retry)<a class="headerlink" href="#crush-msr-multi-step-retry" title="Permalink to this heading"></a></h1>
<section id="motivation">
<h2>Motivation<a class="headerlink" href="#motivation" title="Permalink to this heading"></a></h2>
<p>Conventional CRUSH has an important limitation: rules with
multiple <cite>choose</cite> steps which hit an <cite>out</cite> osd cannot retry
prior steps.  As an example, with a rule like</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">rule</span> <span class="n">replicated_rule_1</span> <span class="p">{</span>
    <span class="o">...</span>
    <span class="n">step</span> <span class="n">take</span> <span class="n">default</span> <span class="k">class</span> <span class="nc">hdd</span>
    <span class="n">step</span> <span class="n">chooseleaf</span> <span class="n">firstn</span> <span class="mi">3</span> <span class="nb">type</span> <span class="n">host</span>
    <span class="n">step</span> <span class="n">emit</span>
<span class="p">}</span>
</pre></div>
</div>
<p>one might expect that if all of the OSDs on a particular host
are marked out, mappings including those OSDs would end up
on another host (provided that there are enough hosts).  Indeed,
that’s what will happen.  Moreover, if 1/8 OSDs on a host are
marked out, roughly 1/8 of the PGs mapped to that host will end
up remapped to some other host keeping overall per-OSD utilization
even.</p>
<p>Suppose, instead, the rule were written like this:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">rule</span> <span class="n">replicated_rule_1</span> <span class="p">{</span>
    <span class="o">...</span>
    <span class="n">step</span> <span class="n">take</span> <span class="n">default</span> <span class="k">class</span> <span class="nc">hdd</span>
    <span class="n">step</span> <span class="n">choose</span> <span class="n">firstn</span> <span class="mi">3</span> <span class="nb">type</span> <span class="n">host</span>
    <span class="n">step</span> <span class="n">choose</span> <span class="n">firstn</span> <span class="mi">1</span> <span class="nb">type</span> <span class="n">osd</span>
    <span class="n">step</span> <span class="n">emit</span>
<span class="p">}</span>
</pre></div>
</div>
<p>The behavior would be very similar as long as no OSDs are marked
out.  However, if an OSD is marked out, any PGs mapped to that
OSD will be remapped to other OSDs on the same host resulting in
those OSDs being over-utilized relative to OSDs on other hosts.
Moreover, if all of the OSDs on a host are marked out, mappings
that happen to hit that host will fail resulting in undersized PGs.</p>
<p>As long as the goal is to split N OSDs between N failure domains,
the solution is simply to use the <cite>chooseleaf</cite> variant above.  However,
consider a use case where we want to split an 8+6 EC encoding over 4
hosts in order to tolerate the loss of a host and an OSD on another
host with 1.75x storage overhead.  The rule would have to look
something like:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">rule</span> <span class="n">ecpool</span><span class="o">-</span><span class="mi">86</span> <span class="p">{</span>
    <span class="o">...</span>
    <span class="n">step</span> <span class="n">take</span> <span class="n">default</span> <span class="k">class</span> <span class="nc">hdd</span>
    <span class="n">step</span> <span class="n">choose</span> <span class="n">indep</span> <span class="mi">4</span> <span class="nb">type</span> <span class="n">host</span>
    <span class="n">step</span> <span class="n">choose</span> <span class="n">indep</span> <span class="mi">4</span> <span class="nb">type</span> <span class="n">osd</span>
    <span class="n">step</span> <span class="n">emit</span>
<span class="p">}</span>
</pre></div>
</div>
<p>This does split up to 16 OSDs between 4 hosts (with an 8+6 code,
it would put 4 OSDs on each of the first 3 and 2 on the last) and
meets our failure requirements.  However, for the reasons outlined
above, it will behave poorly as OSDs are marked out if there are
other hosts to rebalance to.  <cite>chooseleaf</cite> is not a solution here
because it does not support mapping more than one leaf below the
specified type.</p>
</section>
<section id="msr">
<h2>MSR<a class="headerlink" href="#msr" title="Permalink to this heading"></a></h2>
<p>CRUSH MSR (Multi-step Retry) rules solve the above problem by using a
different descent algorithm which retries all of the steps upon
hitting an out OSD.  Where classic CRUSH is breadth first (for each
step, it fully populates the vector before proceeding to the next
step), MSR rules are depth first -- for each choice, we recursively
descend through all of the steps before continuing with the next
choice.  The above use case can be satisfied with the following rule:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">rule</span> <span class="n">ecpool</span><span class="o">-</span><span class="mi">86</span> <span class="p">{</span>
    <span class="nb">type</span> <span class="n">msr_indep</span>
    <span class="o">...</span>
    <span class="n">step</span> <span class="n">take</span> <span class="n">default</span> <span class="k">class</span> <span class="nc">hdd</span>
    <span class="n">step</span> <span class="n">choosemsr</span> <span class="mi">4</span> <span class="nb">type</span> <span class="n">host</span>
    <span class="n">step</span> <span class="n">choosemsr</span> <span class="mi">4</span> <span class="nb">type</span> <span class="n">osd</span>
    <span class="n">step</span> <span class="n">emit</span>
<span class="p">}</span>
</pre></div>
</div>
<p>As with the <cite>chooseleaf</cite> example at the top, as OSDs are marked out,
those OSDs are be remapped proportionately to other hosts so long as
there are extras available.  For details on how that works while
still preserving failure domain isolation, see the comments in
mapper.c:crush_msr_choose.</p>
</section>
<section id="rule-structure">
<h2>Rule Structure<a class="headerlink" href="#rule-structure" title="Permalink to this heading"></a></h2>
<p>CRUSH MSR rules are crush rules with type CRUSH_RULE_TYPE_MSR_FIRSTN
or CRUSH_RULE_TYPE_MSR_INDEP (see mapper.c: rule_type_is_msr).  Unlike
with classic crush rules, individual steps do not specify firstn or
indep.  The output order is instead defined by the rule type for the
whole rule.</p>
<p>MSR rules have some structural differences from conventional rules:</p>
<ul class="simple">
<li><p>The rule type determines whether the mapping is FIRSTN or INDEP.
Because the descent can retry steps, it doesn’t really make sense
for steps to individually specify output order and I’m not really
aware of any use cases that would benefit from it.</p></li>
<li><p>MSR rules <em>must</em> be structured as a (possibly empty) prefix of
config steps (CRUSH_RULE_SET_CHOOSE_MSR*) followed by a sequence of
EMIT blocks each comprised of a TAKE step, a sequence of CHOOSE_MSR
steps, and ended by an EMIT step.</p></li>
<li><p>MSR steps must be <cite>choosemsr</cite>.  <cite>choose</cite> and <cite>chooseleaf</cite> are not
permitted.</p></li>
</ul>
</section>
<section id="working-space">
<h2>Working Space<a class="headerlink" href="#working-space" title="Permalink to this heading"></a></h2>
<p>MSR rules also have different requirements for working space.
Conventional CRUSH requires 3 vectors of size result_max to use for
working space -- two to alternate as it processes each rule and one,
additionally, for <cite>chooseleaf</cite>.  MSR rules need N vectors where N is the
number of <cite>choosemsr</cite> steps in the longest EMIT block since it needs to
retain all of the choices made as part of each descent.</p>
<p>See mapper.h/c:crush_work_size, crush_msr_scan_rule for details.</p>
</section>
<section id="implementation">
<h2>Implementation<a class="headerlink" href="#implementation" title="Permalink to this heading"></a></h2>
<p>mapper.h/c:crush_do_rule internally branches to
mapper.c:crush_msr_do_rule for rules of type CRUSH_RULE_TYPE_MSR_*
(see mapper.c:rule_type_is_msr).</p>
<p>MSR related functions in mapper.c are annotated with more details
about the algorithm.</p>
</section>
</section>



<div id="support-the-ceph-foundation" class="admonition note">
  <p class="first admonition-title">Brought to you by the Ceph Foundation</p>
  <p class="last">The Ceph Documentation is a community resource funded and hosted by the non-profit <a href="https://ceph.io/en/foundation/">Ceph Foundation</a>. If you would like to support this and our other efforts, please consider <a href="https://ceph.io/en/foundation/join/">joining now</a>.</p>
</div>


           </div>
           
          </div>
          <footer><div class="rst-footer-buttons" role="navigation" aria-label="Footer">
        <a href="../cpu-profiler/" class="btn btn-neutral float-left" title="Oprofile 的安装" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
        <a href="../cxx/" class="btn btn-neutral float-right" title="C++17 and libstdc++ ABI" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>&#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).</p>
  </div>

   

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>